Towards the Design of a Natural User Interface for Performing and Learning Musical Gestures

AbstractA large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains a musical gesture. In this paper we present the conceptualization of a Natural User Interface (NUI), named the Intangible Musical Instrument (IMI), aiming to support both learning of expert musical gestures and performing music as a unified user experience. The IMI is designed to recognize metaphors of pianistic gestures, focusing on subtle uses of fingers and upper-body. Based on a typology of musical gestures, a gesture vocabulary has been created, hierarchized from basic to complex. These piano-like gestures are finally recognized and transformed into sounds

Music gestural skills development engaging teachers, learners and expert performers

International audienceThis article presents a platform for learning theoretical knowledge and practical motor skills of musical gestures by combining functionalities of Learning Management Systems (LMS) and Serious Gaming (SG). The teacher designs his/her educational scenario that can be articulated by both theoretical and practical activities. The learner accesses online multimedia courses by using his/her LMS client which can be a computer, tablet orsmartphone and the serious game by using his/her computer and the motion capture sensors. During practicing, his/her gestures are compared in real-time with the expert gestures and s/he is evaluated both in terms of correct fingerings and kinematics. Finally, the platform offers a single profile for the learner for theoretical and practical activities

Using Augmented Reality in K-12 Education: An Indicative Platform for Teaching Physics

Augmented Reality (AR) could provide key benefits in education and create a richer user experience by increasing the motivation and engagement of the students. To this end, the current paper presents a system with three AR applications for teaching physics in the fifth and sixth grades of primary school and in the first grade of secondary school, and the ultimate goal is the development of a unified platform that covers the subject of physics in all classes of K-12 education. The platform provides a useful tool to familiarize both teachers and pupils with AR technologies, aiming to improve the learning and teaching experience and to enhance their skills. The developed system is evaluated in terms of usability, gamification and willingness of the teachers to incorporate this technology into the teaching process. A total of 314 users participated in the research, where they were divided into three user groups: (i) teachers (N = 15), (ii) pupils (N = 189) and (iii) computer science students (N = 110). The outcomes were satisfactory, revealing that the gamified AR applications are easy to use, and teachers are interested in using these AR applications in their classrooms

Machine learning in sonification of expressive gesture with the use of stochastic models

A wider scope of this thesis is to investigate the gestural know-how of a musician and specifically of a pianist, which includes not only the acquisition of theoretical knowledge but also that of practical skills. Emphasis is given, in this thesis, in the expressive gesture and its variations. The gesture recognition is accomplished by using machine-learning algorithms and motion capture technologies. According to the literature review, several research approaches have been conducted aiming not only at the recognition of the performed gesture but also at the sonification of the gesture. However, a weakness that emerges is that the existing methodologies fail to take into account expressive variations that may occur during the performance of the incoming gesture with respect to the template gesture. This results in the recognition not being correct throughout the performance of the expressive gesture and in overlaps occurring between the different classes of classification. As a consequence, the sonification of the expressive gesture is not fluid and immediate.For this purpose, the proposed thesis presents a novel methodology which aims at a) the stochastic modeling, b) the gesture recognition and c) the sonification of the expressive gesture of the user, taking into account possible variations that may occur during the performance of the expressive gesture. This is achieved with the development of the Expert Operational Model, through which the confidence bounds are extracted. The added value of the Expert Operational Model, and thus of the confidence bounds, is that, during the recognition, the system prevents numerical errors that may occur due to variations, made either intentionally or not, and which may also be regarded as expressive elements of the performance of the gesture. The recognition of the expressive gesture is implemented by using machine-learning algorithms and specifically the Particle Filter algorithm. In addition, sound synthesis methods are applied to gesture sonification, providing the user the ability to resynthesize and manipulate the sound continuously and in real-time.The evaluation of the proposed methodology in comparison with established techniques and machine-learning algorithms, shows higher percentages of recognition, accuracy and similarity between the produced sound and the original. Another observation is that the quality of the produced and resynthesized sound in real-time, directly depends on the quality of the recognition of the expressive gesture. The better the performance of the incoming expressive gesture, the better, more fluid and without oscillations is the recognition of the expressive gesture. Hence, the re-synthesis of the sound is better and more fluid. Finally, the positive results of the evaluation, along with the proposed theoretical framework, confirm the efficient use of the confidence bounds in the recognition and sonification of the expressive gesture.Ευρύτερο αντικείμενο της παρούσας διδακτορικής διατριβής αποτελεί η διερεύνηση της χειρονομιακής τεχνογνωσίας του μουσικού και συγκεκριμένα του πιανίστα, η οποία περιλαμβάνει την απόκτηση όχι μόνο θεωρητικών γνώσεων αλλά και πρακτικών δεξιοτήτων. Στη συγκεκριμένη διατριβή, έμφαση δίνεται κυρίως στην εκφραστική χειρονομία και στις διακυμάνσεις της. Η αναγνώριση της χειρονομίας επιτυγχάνεται με τη χρήση αλγορίθμων μηχανικής μάθησης και τεχνολογιών αναγνώρισης της κίνησης. Σύμφωνα με τη βιβλιογραφική επισκόπηση, αρκετές έρευνες που έχουν διεξαχθεί στοχεύουν όχι μόνο στην αναγνώριση της χειρονομίας που εκτελείται από το χρήστη αλλά και στην ηχοποίηση αυτής. Μια αδυναμία όμως που αναδύεται, έγκειται στο ότι οι υπάρχουσες μεθοδολογίες αδυνατούν να λάβουν υπόψη τους πιθανές εκφραστικές διακυμάνσεις και μεταβολές που μπορεί να συμβούν κατά τη διάρκεια εκτέλεσης της εισερχόμενης χειρονομίας του χρήστη σε σχέση με τη χειρονομία πρότυπο του ειδικού. Αυτό έχει ως συνέπεια η αναγνώριση να μην είναι σωστή καθ’ όλη τη διάρκεια εκτέλεσης της εκφραστικής χειρονομίας και να υπάρχουν αλληλοεπικαλύψεις ανάμεσα στις διαφορετικές κλάσεις της ταξινόμησης, με αποτέλεσμα να μην είναι ομαλή και συνεχόμενη και η ηχοποίηση της εκφραστικής χειρονομίας. Για το λόγο αυτό, η παρούσα διδακτορική διατριβή παρουσιάζει μια πρωτότυπη μεθοδολογία που αποσκοπεί α) στη στοχαστική μοντελοποίηση, β) στην αναγνώριση και γ) στην ηχοποίηση της εκφραστικής χειρονομίας του χρήστη, λαμβάνοντας υπόψη τις πιθανές μεταβολές και διακυμάνσεις που μπορεί να συμβούν κατά τη διάρκεια εκτέλεσης της εκφραστικής χειρονομίας. Αυτό επιτυγχάνεται με τη δημιουργία και την ανάπτυξη του Λειτουργικού Μοντέλου του Ειδικού, μέσω του οποίου υπολογίζονται τα όρια εμπιστοσύνης. Η προστιθέμενη αξία του Λειτουργικού Μοντέλου του Ειδικού και κατ’ επέκταση των ορίων εμπιστοσύνης, είναι ότι κατά τη διάρκεια της αναγνώρισης, το σύστημα αποτρέπει αριθμητικά σφάλματα που μπορεί να συμβούν λόγω μεταβολών και διακυμάνσεων, που γίνονται είτε εσκεμμένα είτε όχι, και τα οποία μπορούν επίσης να θεωρηθούν ως εκφραστικά στοιχεία της εκτέλεσης της χειρονομίας. Η αναγνώριση της εκφραστικής χειρονομίας υλοποιείται με τη χρήση αλγορίθμων μηχανικής μάθησης και συγκεκριμένα του αλγορίθμου Φίλτρο Σωματιδίων. Επιπρόσθετα για την ηχοποίηση της εκφραστικής χειρονομίας εφαρμόζονται μέθοδοι σύνθεσης ήχου, παρέχοντας στο χρήστη τη δυνατότητα επανασύνθεσης και χειρισμού του ήχου συνεχόμενα και σε πραγματικό χρόνο. Η αξιολόγηση της μεθοδολογίας μέσω της συγκριτικής μελέτης με άλλους αλγορίθμους γνωστούς στη βιβλιογραφία, έδειξε υψηλότερα ποσοστά αναγνώρισης, ακρίβειας και ομοιότητας ανάμεσα στον παραγόμενο και στον πρωτότυπο ήχο. Παρατηρήθηκε επίσης ότι η ποιότητα του παραγόμενου ήχου που επανασυντίθεται σε πραγματικό χρόνο από τα χειρονομιακά δεδομένα του χρήστη, εξαρτάται άμεσα από την ποιότητα αναγνώρισης των εκφραστικών χειρονομιών. Δηλαδή, όσο καλύτερη είναι η εκτέλεση της εισερχόμενης εκφραστικής χειρονομίας, τόσο καλύτερη, ομαλότερη και χωρίς ταλαντώσεις είναι η αναγνώριση της εκφραστικής χειρονομίας και άρα τόσο καλύτερη και ομαλότερη είναι και η επανασύνθεση του ήχου. Τέλος, τα θετικά αποτελέσματα της αξιολόγησης, σε συνδυασμό με το προτεινόμενο θεωρητικό πλαίσιο, επιβεβαιώνουν ότι είναι αποδοτική η χρήση των ορίων εμπιστοσύνης κατά την αναγνώριση και την ηχοποίηση της εκφραστικής χειρονομίας

Migraine: an invisible disorder : A literature study on non-pharmacological methods in adult patients with migraine

Migrän är en återkommande, anfallsvis huvudvärk som räknas till en av de 20 mest handikappande sjukdomarna i världen. Det är sjuksköterskans ansvar att främja hälsan hos patienter med migrän genom att upplysa om sjukdomen och informera om vilka icke-farmakologiska metoder som kan verka mot migrän. Syftet med studien var att beskriva icke-farmakologiska metoder vid migränanfall hos vuxna. Litteraturstudiens resultat baserades på 13 vetenskapliga artiklar som genererade tre olika teman: akupunktur, kunskap kring utlösande faktorer samt livsstilsförändringar. Resultatet visade att akupunktur, kunskap och utbildning kring utlösande faktorer samt livsstilsförändringar är bra behandlingsalternativ i det hälsofrämjande arbetet vid migrän, då metoderna visade på signifikant minskning av migränanfall. För att bevisa om metoderna är tillförlitliga behövs ytterligare forskning angående icke-farmakologiska metoder vid migrän. Utformning av riktlinjer gällande migrän och adekvata behandlingsmetoder ute i vårdverksamheten vore betydelsefullt. Utökad utbildning om migrän i sjuksköterskeutbildningarna är av stor vikt.Migraine is a recurrent headache that occurs in periods of sudden attacks and is regarded as one of the 20 most disabling disorders in the world. It is the responsibility of the nurse to promote the health of patients with migraine by educating about migraine and by providing vital information about non-pharmacological methods that has beneficial effects on migraine. The aim of this literature study was to describe non-pharmacological methods on adult migraine patients. The result of this literature study was based on 13 scientific articles that generated three themes: acupuncture, knowledge about trigger factors and lifestyle changes. The result showed that acupuncture, knowledge about migraine triggers and lifestyle changes are favourable alternative treatments in migraine health promotion as they demonstrated significant reduction of migraine attacks. To attest its validity further research regarding the use of non-pharmacological methods on migraine is necessary. Specific guidelines regarding migraine and adequate available treatment methods in healthcare settings would be of great importance. We encourage additional education on migraine and nonpharmacological, preventive methods to be integrated in nursing schools

Modelling gestural know-how in pottery based on state-space estimation and system dynamic simulation

International audienceThis paper presents a methodology for modelling expert gestural performances in wheel-throwing pottery. The approach is based on building an operational model that describes how expert gestures are performed, taking also into consideration relationships between different parts of the body. This model is estimated using state-space estimation methodology and its predictive performance is evaluated using system dynamic simulation

Offline Statistical Analysis of Gestural Skills in Pottery Interaction

International audienc

x2Gesture: how machines could learn expressive gesture variations of expert musicians

International audienceThere is a growing interest in ‘unlocking’ the motor skills of expertmusicians. Motivated by this need, the main objective of this paper isto present a new way of modeling expressive gesture variations inmusical performance. For this purpose, the 3D gesture recognitionengine ‘x2Gesture’ (eXpert eXpressive Gesture) has been developed,inspired by the Gesture Variation Follower, which is initially designedand developed at IRCAM in Paris and then extended at GoldsmithsCollege in London. x2Gesture supports both learning of musicalgestures and live performing, through gesture sonification, as a unifieduser experience. The deeper understanding of the expressive gesturalvariations permits to define the confidence bounds of the expert’sgestures, which are used during the decoding phase of the recognition.The first experiments show promising results in terms of recognitionaccuracy and temporal alignment between template and performedgesture, which leads to a better fluidity and immediacy and thusgesture sonification

Modelling Gestural Know-how in Pottery Based on State-space Estimation and System Dynamic Simulation

AbstractThis paper presents a methodology for modelling expert gestural performances in wheel-throwing pottery. The approach is based on building an operational model that describes how expert gestures are performed, taking also into consideration relationships between different parts of the body. This model is estimated using state-space estimation methodology and its predictive performance is evaluated using system dynamic simulation. Moreover, the confidence bounds derived of the expert gesture performance in wheel-throw pottery are computed. They could be used as benchmarks in real-time experiments in order to generate the appropriate feedback for sensorimotor learning of gestures

HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument

In this paper, an interactive learning experience is proposed, aiming to involve museum visitors in a personalized experience of the transmittal of cultural knowledge in an active and creative way. The proposed system, called HapticSOUND, consists of three subsystems: (a) the Information, where visitors are informed about the traditional musical instruments; (b) the Entertainment, where visitors are entertained by playing serious games to virtually assemble traditional musical instruments by a set of 3D objects; and (c) the Interaction, where visitors interact with a digital musical instrument which is an exact 3D-printed replica of a traditional musical instrument, where cameras have been placed to capture user gestures and machine learning algorithms have been implemented for gesture recognition. The museum visitor can interact with the lifelike replica to tactilely and aurally explore the instrument’s abilities, producing sounds guided by the system and receiving real-time visual and audio feedback. Emphasis is given to the Interaction Subsystem, where a pilot study was conducted to evaluate the usability of the subsystem. Preliminary results were promising since the usability was satisfactory, indicating that it is an innovative approach that utilizes sensorimotor learning and machine learning techniques in the context of playing sounds based on real-time gesture and fingering recognition